System to add parallax to video for augmented reality head up display

ABSTRACT

A head up display arrangement for a motor vehicle includes a source of a first video signal. A driver monitor system detects a location of at least a portion of a human driver of the motor vehicle. An electronic processor is communicatively coupled to the first video signal source and to the driver monitor system. The electronic processor produces a second video signal that is dependent upon the first video signal and that includes parallax information. The parallax information is based on the detected location of the portion of the driver. A head up display projection device produces a virtual image that is visible to the driver and that is based upon the second video signal.

CROSS-REFERENCED TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Application No.62/847,537 filed on May 14, 2019, the disclosure of which is herebyincorporated by reference in its entirety for all purposes.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a head up display (HUD) system in amotor vehicle.

2. Description of the Related Art

A head up display emits light that reflects off of one or more mirrorsand from the front windshield to be seen by the driver. The lightappears to come from a virtual image in front of the driver and in frontof the windshield. This type of head up display is currentlycommercially available.

Conventional head up displays create the virtual image by first using adisplay to create an image. Next, the light from the image is reflectedfrom one or more mirrors. Next, the light from the mirrors is reflectedfrom the windshield. The mirrors are designed and positioned relative tothe display so that the light seen by the driver, which is reflectedfrom the windshield, appears to come from a virtual image that isoutside of the vehicle. The mirrors and display are typically containedin a package that occupies a volume beneath the top surface of thedashboard.

SUMMARY OF THE INVENTION

The invention may provide a means to position graphics to be viewed inan augmented reality (AR) head up display so the driver has theperception of parallax of the virtual objects, even though the inputvideo does not include the effect of parallax. If the HUD is providedwith the distance from the driver to the virtual object, the image canbe corrected for parallax. The HUD does not need this information foreach individual point. The HUD may divide the field of view into zones,similar to what is done with local dimming of the backlight, and applythe parallax shift individually to each zone. This calls for the vehiclevideo control unit (VCU) to send the distance for each zone. The numberof zones may be small enough that the processing demands are reasonable.For example, if the field of view were divided into a 6×8 matrix of 48zones, the effect may be acceptable from a human factors point of view.To implement this, the VCU would need to send only 48 distances witheach video frame, which would be reasonable for the HUD to implement.

The invention comprises, in one form thereof, a head up displayarrangement for a motor vehicle, including a source of a first videosignal. A driver monitor system detects a location of at least a portionof a human driver of the motor vehicle. An electronic processor iscommunicatively coupled to the first video signal source and to thedriver monitor system. The electronic processor produces a second videosignal that is dependent upon the first video signal and that includesparallax information. The parallax information is based on the detectedlocation of the portion of the driver. A head up display projectiondevice produces a virtual image that is visible to the driver and thatis based upon the second video signal.

The invention comprises, in another form thereof, a method forpresenting visual information to a human driver in a motor vehicle. Afirst video signal is transmitted. A location of at least a portion ofthe human driver of the motor vehicle is detected. A second video signalthat is dependent upon the first video signal and that includes parallaxinformation is transmitted. The parallax information is based on thedetected location of the portion of the driver. A virtual image that isvisible to the driver and that is based upon the second video signal isproduced.

The invention comprises, in yet another form thereof, a head up displayarrangement for a motor vehicle, including a source of a first videosignal. A driver monitor system emits infrared energy through a dichroicmirror such that the infrared energy is reflected off of a human driverof the motor vehicle. The driver monitor system receives the reflectedinfrared energy after the infrared energy is reflected off of a humandriver. The driver monitor system detects a location of at least aportion of a human driver of the motor vehicle based upon the receivedinfrared energy. An electronic processor is communicatively coupled tothe first video signal source and to the driver monitor system. Theelectronic processor produces a second video signal that is dependentupon the first video signal and that includes parallax information. Theparallax information is based on the detected location of the portion ofthe driver. A head up display projection device emits a light field thatis reflected off of the dichroic mirror to thereby produce a virtualimage that is visible to the driver and that is based upon the secondvideo signal.

An advantage of the invention is that it makes it possible for anaugmented reality HUD to correctly provide the perception of parallax tothe driver even though eye position information is not provided to thevehicle video control unit.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features and objects of this invention,and the manner of attaining them, will become more apparent and theinvention itself will be better understood by reference to the followingdescription of embodiments of the invention taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 is a block diagram of one embodiment of an AR HUD system of thepresent invention.

FIG. 2 is a schematic diagram of an AR HUD arrangement including the ARHUD system of FIG. 1.

FIG. 3 is a plan view of one embodiment of a field of view of a virtualimage of the present invention.

FIG. 4 is a block diagram of one embodiment of an AR HUD visionprocessing arrangement of the present invention.

FIG. 5 is a schematic diagram illustrating the relationship between theeyebox and the virtual image according to one embodiment of the presentinvention.

FIG. 6 is a flow chart of one embodiment of a method of the presentinvention for presenting visual information to a human driver in a motorvehicle.

DETAILED DESCRIPTION

The embodiments hereinafter disclosed are not intended to be exhaustiveor limit the invention to the precise forms disclosed in the followingdescription. Rather the embodiments are chosen and described so thatothers skilled in the art may utilize its teachings.

FIG. 1 illustrates one embodiment of an AR HUD system 10 of the presentinvention including a projector 12, a driver monitor system 14, anelectronic processor 16, and a video signal source in the form ofvehicle video control unit (VCU) 18. Projector 12 emits visible light 20which may be seen by a human driver 22 as a virtual image with augmentedreality.

Driver monitor system 14 may determine the position of the driver's eyepoint. That is, driver monitor system 14 may determine the position orlocation of one or both eyes 24 of driver 22 in three-dimensional space.In order to accomplish this, driver monitor system 14 may provideinfrared illumination 26, and receive infrared energy 28 that has beenreflected off of driver 22.

As indicated at 30, processor 16 may receive from driver monitor system14 information indicating the location of one or both eyes 24 of driver22 in three-dimensional space. Based on the location of one or both eyes24, processor 16 may include or add parallax information in a videosignal 32 that is transmitted to projector 12.

Vehicle video control unit 18 may transmit a video stream or signal 34of graphics without parallax to processor 16. Vehicle video control unit18 may transmit another signal 36 indicating the perceived distancebetween driver 22 and virtual objects in a sequence of zones in thedriver's field of view.

The graphics to be shown may be created within video control unit 18.VCU 18 may send the graphics as an input video stream to the HUD,including projector 12 and processor 16. The input video stream to theHUD does not include the effect of parallax. Packaged with the HUD isdriver monitor system (DMS) 14 which determines a point in spacecharacteristic of the location of the driver's eyes 24. According to oneembodiment of the invention, the field of view of the HUD is dividedinto zones, and VCU 18 also sends to the HUD a stream of information, byzones, of average distance from the driver to the virtual objects inthat zone.

The AR HUD may input video (without parallax) from the vehicle videocontrol unit 18 together with an information stream including thedistances to virtual objects in each of a sequence of zones in thevirtual image. The AR HUD may also output a projected image to be seenby the driver as augmented reality with parallax. The location of apoint characteristic of the driver's eyes is obtained using drivermonitor system 14, which illuminates the driver's face with infraredenergy and which images the infrared energy reflected off of thedriver's face. Processor 16 combines the video signal from the vehicleVCU 18, the distance information to the virtual object, and the driver'seye point to provide a video stream to projector 12 that includesparallax.

For zones in which there is no graphical content, a default value isprovided in the stream. For each zone, the HUD calculates thedisplacement of the graphics based on driver eye position, distance tothe virtual object, and the virtual image distance. As an example, thegraphical content is translated accordingly before the graphical contentis displayed. As an example, in a particular zone, the virtual image istranslated from 2D position d to position d′ where:d′=d(OD−VID)/OD.wherein OD is the distance from the driver to the virtual object, andVID is the distance from the driver to the virtual image. The origin ofd is chosen so that with d=0, points in the virtual image are alignedwith corresponding points in the real world. Each point in the virtualimage has a corresponding pixel in the display that illuminates thatpoint. To express the function of this invention in terms of individualpixels on the HUD display, it is necessary to take the mapping from theHUD display to the virtual image into account, as is known to the art.

FIG. 2 illustrates an AR HUD arrangement 100 of FIG. 1 including AR HUDsystem 10. Projector 12 may be in the form of a HUD picture generationunit. AR HUD arrangement 100 further includes vehicle sensors andsystems 38, a dichroic mirror 40, a freeform mirror 42, and a vehiclewindshield 44.

Visible light from HUD picture generation unit 12 reflects from dichroicmirror 40, reflects from freeform mirror 42, and reflects from thewindshield 44 to be seen by the driver as a virtual image 46 outside ofwindshield 44.

Infrared light 48 from driver monitor system 14 passes through dichroicmirror 40, reflects from freeform mirror 42, and reflects fromwindshield 44 to illuminate the driver's face. Reflected IR from thedriver's face retraces the same path: reflects from windshield 44,reflects from freeform mirror 42, and passes through dichroic mirror 40to be imaged by a camera (not shown) in driver monitor system 14. Drivermonitor system 14 outputs the eye position to processor 16.

Vehicle video control unit 18 creates augmented reality graphics thatdoes not take the driver's head position into account. The video fromvehicle video control unit 18 passes to processor 16, along withinformation that describes the average distance from driver 22 to theaugmented reality object(s) in various zones in virtual image 46.Processor 16 combines the input video, the driver eye position, and theaverage distance from driver 22 to the virtual object(s) to produce avideo stream to HUD picture generation unit 12 that correctly includesparallax.

In an alternate embodiment (not shown), the driver monitor system ispositioned behind the freeform mirror, which has a dichroic coating. Theflat dichroic mirror shown in FIG. 2 is removed, and the HUD picturegeneration unit is placed in the position of driver monitor system as itis shown in FIG. 2.

In one embodiment, the driver's head is tracked in sequential frames andthe collected data is used to predict the driver's head position,thereby providing an improved virtual image and avoiding a perception oflatency. The relative velocity of the driver's eye point is calculatedfrom the measured change in driver's eye point between two or morepoints, and the frame rate at which the camera shutter operates.

In one embodiment, let T1 and T2 be two sequential times at which theDMS system measures d1 and d2, respectively, as the two-dimensionalvector characteristic of the driver's transverse eye position. Let OD1and OD2 be the distance from the driver to the virtual object at timesT1 and T2, respectively. Let T2 be the time of the most recentmeasurement, and T3 be the time at which the image will be presented tothe driver. The transverse velocity of the eye position is:VT=(d2−d1)/(T2−T1). The longitudinal velocity of the eye position is:VOD=(OP2−OP1)/(T2−T1). The predicted value of d at time T3 is:d3=d2+(T3−T2)*VT. The predicted value of OD at time T3 is:OD3=OD2+VOD*(T3−T2).

The processor uses d3 and OP3 to update the appearance of the virtualobject (instead of d2 and OP2) to provide the perception that thevirtual object is locked to the scene in front of the driver. It ispossible to use the measured head position at more than two sequentialtimes to estimate velocity, so as to minimize the driver's perception ofjitter in the virtual image.

As an example of how the invention can be implemented, let the field ofview of the virtual image be rectangular, with angular dimensions H×V.As an example, H is divided into N zones and V is divided into M zones,where N and M are integers >0, as shown in FIG. 3, and where N=19 andM=7. The average distance to the virtual objects in each zone isdetermined. For example, the average distance to the virtual objects inzone(J, K) is OP(J, K). Then, for pixels in zone(J, K), the value of OPused in Eq. (1) is OP(J, K).

FIG. 3 illustrates how the field of view is divided into zones. Theaverage object distance is determined for each zone. The average objectdistance for each zone is communicated from the video control unit tothe processor, and is used in the determination of the transformed imagefor that zone. The field of view and the zones do not need to berectangular.

In one embodiment, in zone(J, K), for example, there are QP total pixelsthat display a virtual object. Let QPL be an index that labels thepixels from 1 to QP. In one embodiment, the average value OP(J, K)=[OP(QPL=1)+ . . . +OP (OPL=QP)]/QP. In a second embodiment, the points areweighted by (1/distance) in computing the average, to conform to theeye's perception. Thus, the average value:OP(J,K)=1/{[1/OP(QPL=1)+ . . . +1/OP(QPL=QP)]/QP]}

As a specific embodiment, this invention can be implemented using an NXPS32V vision processor as shown in FIG. 4. The NXP S32V device isavailable from NXP Semiconductor of Eindhoven, The Netherlands. FIG. 4illustrates an AR HUD vision processing arrangement 200 includingprocessing cores 250, an image processing block 252 for the drivermonitoring system (DMS), a neural network accelerator 254, a graphicalprocessing unit 256, a data transfer bus 258, video input block 260, avideo output block 262, and an external memory 264. Image processingblock 252 receives data 266 from an IR camera (not shown). As indicatedat 268, video input block 260 receives video input from the VCU anddistance information on a back-channel. Video output block 262 transmitsvideo output 270 to a HUD picture generation unit (not shown).

AR HUD vision processing arrangement 200 inputs video and distanceinformation from the vehicle VCU, and outputs video to the HUD picturegeneration unit. In the example embodiment shown, the image processingfor the infrared camera and the processing to determine the location ofa point characteristic of the driver's eyes is also integrated. In oneembodiment, the system shown is implemented with an NXP S32V visionprocessing device.

An alternative to the present invention is to use the driver eyeposition information in the process of creating the augmented realitygraphics. A benefit of the present invention is that it accommodatesexisting electrical architectures. Moreover, in one application, thecontent is pre-recorded video, and thus the eye position informationmust be added at the display according to the present invention.

The inventive process of adding parallax to a video stream may be usedfor other display applications, not just for an augmented reality headup display. For example, it may be used for entertainment video, such asmovies, to make it appear that near and far objects move as expected asthe viewer moves their head. It may also be used with a transparentdisplay to show virtual objects behind the transparent display thatexhibit the appropriate parallax effect as the viewer moves their head.

The invention may provide compensation within the HUD for the driver'seye position. With a given HUD architecture, the invention enables theHUD itself to compensate for the effect of driver eye position, ratherthan the eye position being compensated for during the creation of thegraphics.

The relationship between the driver eye position and the perceivedindicator position at the object is illustrated in FIG. 5. Assume thatat first the driver's eye is at the center of an eyebox 572 at pointEP1, and a point 574 is created at virtual image 576 so it appears atthe correct location relative to a real object at distance ObjectDistance. Next assume that the driver moves his head such that thedriver's eye is displaced by vector d1 in eyebox 572. The projectedpoint 574 at virtual image 576 stays fixed. The virtual point appears tobe displaced by vector d2 in the plane of the projected virtual image578.

To compensate, so the virtual object appears to be at the correctlocation given the new eye location, the projected point in the virtualimage needs to move so at the object distance, the virtual point isdisplaced −d2. With reference to the diagram of FIG. 5:d2=−d1(Object Distance−VID)/VIDThe displacement on the virtual image to reverse this displacement is:d3=−d2(VID)/(Object Distance)=d1(Object Distance−VID)/(Object Distance)

This approach works if there is only one virtual object, or if allvirtual objects are at the same distance. There is a need to know whatthe Object Distance is. If there are multiple virtual objects atdifferent distances, the compensation can only be accurate for oneObject Distance.

FIG. 6 illustrates one embodiment of a method 600 of the presentinvention for presenting visual information to a human driver in a motorvehicle. In a first step 602, a first video signal is transmitted. Forexample, vehicle video control unit 18 may transmit a video stream orsignal 34 of graphics without parallax to processor 16.

Next, in step 604, a location of at least a portion of the human driverof the motor vehicle is detected. For example, driver monitor system 14may determine the position or location of one or both eyes 24 of driver22 in three-dimensional space.

In a next step 606, a second video signal that is dependent upon thefirst video signal and that includes parallax information istransmitted. The parallax information is based on the detected locationof the portion of the driver. For example, based on the location of oneor both eyes 24, processor 16 may include or add parallax information ina video signal 32 that is dependent upon signal 34 and that istransmitted to projector 12.

In a final step 608, a virtual image that is visible to the driver andthat is based upon the second video signal is produced. For example,visible light from HUD picture generation unit 12, and that is basedupon signal 32, reflects from dichroic mirror 40, reflects from freeformmirror 42, and reflects from the windshield 44 to be seen by the driveras a virtual image 46 outside of windshield 44.

While this invention has been described as having an exemplary design,the present invention may be further modified within the spirit andscope of this disclosure. This application is therefore intended tocover any variations, uses, or adaptations of the invention using itsgeneral principles. Further, this application is intended to cover suchdepartures from the present disclosure as come within known or customarypractice in the art to which this invention pertains.

What is claimed is:
 1. A head up display arrangement for a motorvehicle, comprising: a source of a first video signal; a driver monitorsystem configured to detect a location of at least a portion of a humandriver of the motor vehicle; an electronic processor communicativelycoupled to the first video signal source and to the driver monitorsystem, the electronic processor being configured to produce a secondvideo signal that is dependent upon the first video signal and thatincludes parallax information, the parallax information being based onthe detected location of the portion of the driver, wherein the parallaxinformation is dependent upon an estimate of where the portion of thehuman driver will be at a future point in time; and a head up displayprojection device configured to produce a virtual image that is visibleto the driver and that is based upon the second video signal.
 2. Thearrangement of claim 1, wherein the first video signal lacks parallaxinformation.
 3. The arrangement of claim 1, wherein the driver monitorsystem transmits infrared energy toward the driver and detects theinfrared energy after the infrared energy has been reflected off of thedriver.
 4. The arrangement of claim 1, wherein the driver monitor systemis configured to detect a location of an eye of a human driver.
 5. Thearrangement of claim 1, wherein the virtual image includes a pluralityof zones, the parallax information being different for each of thezones.
 6. The arrangement of claim 5, wherein the second video signal isdependent upon a plurality of perceived distances between the driver andrespective virtual objects in respective ones of the zones.
 7. Thearrangement of claim 6, wherein a respective position of each of thevirtual objects in the virtual image that is based upon the second videosignal is translated relative to a corresponding position of the virtualobject in a hypothetical virtual image that is based upon the firstvideo signal.
 8. The arrangement of claim 1, wherein the parallaxinformation is dependent upon a perceived position of the virtual image.9. A method for presenting visual information to a human driver in amotor vehicle, the method comprising: transmitting a first video signal;detecting a location of at least a portion of the human driver of themotor vehicle; predicting where the portion of the human driver will beat a future point in time; transmitting a second video signal that isdependent upon the first video signal and that includes parallaxinformation, the parallax information being based on the detectedlocation of the portion of the driver and the prediction of where theportion of the human driver will be at a future point in time; andproducing a virtual image that is visible to the driver and that isbased upon the second video signal.
 10. The method of claim 9, whereinthe first video signal lacks parallax information.
 11. The method ofclaim 9, wherein the detecting step includes transmitting infraredenergy toward the driver and sensing the infrared energy after theinfrared energy has been reflected off of the driver.
 12. The method ofclaim 9, wherein the detecting step includes detecting a location of aneye of the human driver.
 13. The method of claim 9, wherein the virtualimage includes a plurality of zones, the parallax information beingdifferent for each of the zones.
 14. The method of claim 13, wherein thesecond video signal is dependent upon a plurality of perceived distancesbetween the driver and respective virtual objects in respective ones ofthe zones.
 15. The method of claim 14, wherein a respective position ofeach of the virtual objects in the virtual image that is based upon thesecond video signal is translated relative to a corresponding positionof the virtual object in a hypothetical virtual image that is based uponthe first video signal.
 16. The method of claim 9, wherein the parallaxinformation is dependent upon a position of the virtual image asperceived by the driver.
 17. The method of claim 9, wherein the parallaxinformation is dependent upon an estimate of where a portion of a headof the human driver will be at a future point in time.
 18. A head updisplay arrangement for a motor vehicle, comprising: a source of a firstvideo signal; a dichroic mirror; a driver monitor system configured to:emit infrared energy through the dichroic mirror such that the infraredenergy is reflected off of a human driver of the motor vehicle; receivethe reflected infrared energy after the infrared energy is reflected offof a human driver; and detect a location of at least a portion of ahuman driver of the motor vehicle based upon the received infraredenergy; an electronic processor communicatively coupled to the firstvideo signal source and to the driver monitor system, the electronicprocessor being configured to produce a second video signal that isdependent upon the first video signal and that includes parallaxinformation, the parallax information being based on the detectedlocation of the portion of the driver and an estimate of where theportion of the human driver will be at a future point in time; and ahead up display projection device configured to emit a light field thatis reflected off of the dichroic mirror to thereby produce a virtualimage that is visible to the driver and that is based upon the secondvideo signal.
 19. The arrangement of claim 18, wherein the first videosignal lacks parallax information.
 20. The arrangement of claim 18,wherein the driver monitor system is configured to detect a location ofan eye of a human driver.